Incident Report: Critical Low Balance and Low Balance Alerts on All Chains
Date: 2023-12-02
Time: 04:45 (GMT+5:30)
Duration: 15 hours
Description
Critical low balance and low balance alerts on all chains for MerkleFunderDepository contracts.
Root Cause
Because of executing the withdrawAll() tx to send all MFD funds to the new MFD contracts.
Impact
Mass critical alerts on all chain which created unnecessary panic.
Timeline
- 04:45 - First noticed the issue.
- 05:00 - Initial diagnosis.
- 06:55 - Started the fix.
- 22:00 - Issue resolved.
Lessons Learned
Actions that will be triggering the critical alerts better be informed beforehand.
Actions Taken
Wallet-watcher was updated and redeployed within 24 hours.
Related Images/Logs
- Escalation link.
Incident Reviewer(s)
- Mertcan Karik
- Ashar Shahid
- Santiago Aceñolaza
- Burak